Method and system for evaluating skills of contestants in online coding competitions

ABSTRACT

This invention relates to a method and apparatus for providing a coding competition. In one embodiment, a method for providing a coding competition includes transmitting a coding problem to contestants, receiving computer code created by a contestant, determining a response of the computer code to test data, and evaluating the response of the computer code to the test data. In another embodiment, a method for evaluating a skill level of a contestant includes electronically communicating a coding problem to contestants, electronically receiving a software program in response to the coding problem from one of the contestants, evaluating the received software program, awarding points to the contestant based on the received software program, and determining a rating for the contestant for the competition based on the number of points awarded to the contestant.

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. provisional patentapplication serial No. 60/260,610, filed Jan. 9, 2001.

TECHNICAL FIELD

[0002] This invention relates to computer-based methods and systems forcompeting and, more particularly, to methods and systems providingcoding competitions.

BACKGROUND INFORMATION

[0003] In the United States and elsewhere, computers have become part ofpeople's everyday lives, both in the workplace and in personalendeavors. This is because a general purpose computer can be programmedto run a variety of software programs each providing differentprocessing and networking functions. Computer programmers developcomputer code. Some companies hire large numbers of computer programmersto develop code on the company's behalf.

[0004] Like many other activities, coding requires a collection ofskills. For example, some coders are extremely skilled, and, forexample, are remarkably accurate and efficient. Professionalcertifications provide information about the base skills possessed by anindividual, but generally there is no objective way to easilydistinguish highly-skilled coders from average coders. This lack ofobjective measure means that highly-skilled coders may not becompensated for their true worth. In addition, in the United States andelsewhere, workers considered to be “professionals,” such as lawyers,accountants, doctors, and managers, command higher salaries than thegeneral population. Meanwhile, highly skilled programmers, who typicallyhave good education, significant responsibility, and relatively uniqueabilities, are treated like production and administrative staff, ratherthan professionals. This is also true for computer coders in otherdisciplines (e.g. hardware design, and other engineering disciplines).

[0005] In the hiring process, firms typically subject coders to aninterview process that may even include some actual testing ofcandidates' coding skills. But it is very difficult for a company, evenwith some skills testing, to definitively identify those programmers whohave the greatest skills, are the most accurate and efficient, and soon. This makes it difficult for a company that wants to hire premiercoders to identify and hire them, and it makes it more difficult forpremier coders to be hired at the compensation levels that they maydeserve.

SUMMARY OF THE INVENTION

[0006] A need therefore exists for mechanisms for skilled coders tobuild marketability and raise their income to reflect the value theirwork product creates. In general, in one aspect of the invention,systems and methods for providing coding competitions address these andother goals. Through the provision of computer-based codingcompetitions, successful coding competition participants can bedistinguished from other coders, which benefits both the coders andtheir employers. Recognizably better coders can command greatersalaries, and employers have an objective means to compare candidatesand find good coders. In some embodiments, good coders have incentivesto participate as competitions provide one or more of entertainment,awards, and future employment opportunities.

[0007] In some embodiments, entities that wish to gain access to skilledcoders sponsor the coding competitions. In some embodiments, a ratingsystem is used to provide a metric that can be used to evaluate aprogrammer's proven skills, knowledge, and ability.

[0008] In various embodiments, coding competitions are provided“on-line,” in the sense that they are provided using computers thatcommunicate over a network. Coding competition participants, alsoreferred to as contestants, use a client computer that communicates witha competition server. The contestants may be located anywhere in theworld if (for example) the network is a global network such as theInternet, or the coding competition participants may be located nearby,even in the same room, if (for example) the network is a local areanetwork.

[0009] In some embodiments, as part of a coding competition, a servercomputer provides to one or more contestants a computer coding problem(e.g. a test question) that requires each contestant to develop computercode. The computer code may be, for example, an entire computerapplication program or a large or small portion of a computerapplication program (e.g. a subroutine or a subsystem, one or moreobjects, etc.). The code may be written in a specific programminglanguage (e.g. C++, Java, C^(#), Fortran, per, etc.), or may be coded inone of a choice of programming languages or coding formats.

[0010] The coding problem may require design and or analysis of theproblem in addition to coding. The computer code may also be forsomething other than a computer application program; for example, it maybe a design for a computer program, or it may be an electronic circuitdesign or implementation, or for another engineering design and/orimplementation. The computer code may be a model that will be forwardengineered into a machine readable format for testing purposes. The codedeveloped by a contestant is generally referred to as computer code,regardless of whether it is all or a portion of an application program,or something else.

[0011] In some embodiments, portions of the coding competitions takeplace in virtual “rooms” in which two or more contestants compete. Eachcontestant in a virtual room receives a coding problem or problems,which problems may be of varying difficulty. Contestants write computercode in response to one or more of the presented coding problem(s). Whena contestant's code is ready, the respective client computer transmitsthe contestant's computer code to a server for evaluation. The time thecontestant took to complete the computer code may be recorded.

[0012] In some embodiments, the server evaluates the computer code, andassigns a number of points to the contestants' submission. In someembodiments, the competition includes multiple rounds in which theserver evaluates contestant computer code for each round. The server mayalso determine a skill level rating or ratings for the contestant. Thedetermination of skill level of contestants may, for example, result inthe awarding of a prize to a contestant, present employmentopportunities to the contestant, attract sponsorship for thecompetition, and present hiring opportunities to the sponsors of thecompetition.

[0013] In some embodiments, contestants submit the requested computercode responsive to the computer coding problems in a first phase of thecompetition, and the response time of the contestants is recorded. Thecomputer code written by the contestants during the first phase of thecompetition is verified during a subsequent phase of the competition.The submitted code can be verified using predetermined test data asinput. In addition, each contestant may have an opportunity to reviewthe computer code submitted by other contestants, and to submit testdata to be provided as input to the other contestants' computer code forverification of that code. In this way, contestants can challenge thework of the other contestants.

[0014] In one embodiment, one or more competitions are held amongprogrammers in a forum setting, analogous to a spectator sport. Thecontestants are located near each other, and spectators can observe thecompetition, and watch the programmers in action. There are displays andother aids to enhance the spectator enjoyment of the competition. Inanother embodiment, in which the competition takes place via a globalcomputer network such as the Internet, the competitions take place in a“virtual forum” setting, in which spectators can observe, via thecomputer network, the activity in the virtual room or rooms. In someembodiments, spectators (either locally or via the network) can not onlyobserve a coding competition, but can participate, by submitting testdata to be used as input to the competitors' submissions.

[0015] In one aspect of the invention, a method for providing a codingcompetition includes the steps of transmitting a coding problem tocontestants over a communications network and receiving computer codecreated by one or more of the contestants in response to the transmittedcoding problem over the communications network. The method also includesthe steps of determining a response of the received computer code totest data submitted by a party other than the contestant and evaluatingthe determined response, where that party may be, for example, a systemadministrator or server, a judge, another contestant, a competitionspectator, and so forth.

[0016] In one embodiment, the method includes the step of providing thetest data as input to reference code and comparing output of thereference code to output of the received computer code. In oneembodiment, the received computer code may be evaluated based on suchfactors as the correctness of the determined the response, the accuracyof the determined response, and the speed of the determined response.Points may be awarded to a contestant based on the evaluation of thedetermined response of the computer code to the test data. A rating mayalso be assigned to the contestant based on the points awarded, and alsobased on prior competition performance. In one embodiment, a contestantis assigned to a division, based on the contestant's rating. Forexample, the contestant may be assigned to one of a first division forcontestants who have previously attained a first rating or greater, anda second division for contestants who have a rating substantially equalto zero or a rating below a predetermined division rating. There may bea third, fourth, and divisions, etc., depending on the number ofcontestants and the spread in skill level. The use of divisions keepsthe competition interesting to the contestants.

[0017] In another aspect, a method for evaluating a skill level of acontestant includes the steps of electronically communicating a codingproblem to contestants during a competition and electronically receivingcomputer code in response to the coding problem from one of thecontestants during the competition. The method also includes the stepsof automatically evaluating the received computer code and awardingpoints to the contestant based on the received computer code. The methodadditionally includes determining a rating for the contestant for thecompetition based on the number of points awarded to the contestant.

[0018] In some embodiments, the method includes determining an averagerating for the competition based on previous ratings of the competitionand a total number of contestants receiving the coding problem. Themethod can also include determining a competition factor for thecompetition based on previous ratings of the contestants, a total numberof contestants receiving the coding problem, and volatility of theratings of the contestants. In another embodiment. the method includesdetermining a maximum rating increase for the contestant. In yet anotherembodiment, the method includes estimating a probability of awarding acontestant more points than other contestants in the competition. Thedetermination of a rating for the contestant may also include adjustingof the rating in smaller increments as frequency that the contestant hasbeen rated increases.

[0019] In yet another aspect of the invention, a method for providingcontestant information to a potential employer includes the steps ofreceiving contestant information from contestants for entry into on-linecoding competitions and providing an on-line coding competition betweencontestants. The method also includes assigning a rating to eachcontestant based on performance in the on-line coding competition andproviding some or all of the contestant information and the assignedratings of one or more contestants to a potential employer of thecontestant. In one embodiment, the method also includes awarding pointsto the contestant based on performance in the on-line competition. Thecontestant information may include such information as a home address,phone number, work address, e-mail address, professional experience,work history, and certain demographic information of the contestant. Inanother embodiment, a rating is assigned to a contestant based on thepoints awarded to the contestant.

[0020] In another aspect, a method for awarding a prize to a contestantincludes providing an on-line coding competition among contestants andawarding a point value to each contestant based on performance in thecompetition. The method also includes awarding prizes to a subset of thecontestants who have been awarded a point value above a predeterminedprize point value. The prize awarded depends on the points awarded withrespect to the predetermined point value. The prize can include, but maynot be limited to a monetary award, an interview with a prospectiveemployer, a tangible good, a job offer, or some combination.

[0021] In another aspect, a method for advertising in a codingcompetition includes receiving contestant information from contestantsfor entry into on-line coding competitions. The method also includes thesteps of providing one or more on-line coding competitions betweencontestants and providing each contestant with one or more advertisementfor viewing during the competition. The method additionally includestracking the viewing time of each advertisement by each contestant andproviding contestant information and the tracked viewing time of theadvertisement to an advertiser.

[0022] In some embodiments, the contestant information is provided tothe advertiser in the aggregate. For instance, the server provides theinformation that twenty coders having a rating above a predeterminednumber viewed the advertisement for a total time of thirty minutes. Inother embodiments, the server provides advertisement to contestantsbased on contestant information. For instance, the server may provide anadvertisement to contestants having Java programming experience whileproviding another advertisement to contestants having C++ programmingexperience.

[0023] In one aspect, an apparatus for providing a coding competitionincludes a web server communicating with a web browser. The web browseris used by a contestant to receive a competition request and enable thecontestant to enter the coding competition using client software. Theapparatus also includes a client interface server communicating with theclient software. The client interface server enables transmission of acoding problem to the client software and reception of computer code inresponse to the coding problem. The apparatus additionally includes anapplication server in communication with the web server and the clientinterface server. The application server interprets client requests,competition requests, and/or the computer code. The apparatus alsoincludes a contest server in communication with the application server.The contest server compiles the computer code submitted by contestants,executes the computer code, tests the computer code, and/or computes arating for the contestant.

BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In the drawings, like reference characters generally refer to thesame parts throughout the different views. Also, the drawings are notnecessarily to scale, emphasis instead generally being placed uponillustrating the principles of the invention.

[0025]FIG. 1 is a block diagram of an embodiment of a coding competitionsystem according to the invention.

[0026]FIG. 2 is a block diagram of an embodiment of a server in thecoding competition system of FIG. 1.

[0027]FIG. 3 is a more detailed block diagram of an embodiment of theserver of FIG. 2.

[0028]FIG. 4 is a flowchart of the operation of an embodiment of theserver for a coding competition.

[0029]FIG. 5 is a more detailed flowchart of the operation of anembodiment of the server for a coding competition.

[0030]FIG. 6 is a block diagram of an embodiment of the evaluationperformed by the server of a computer code's response to test data.

[0031]FIG. 7 is a flowchart of the operation of an embodiment of theserver for a qualifying test before the coding competition of FIG. 4.

[0032]FIG. 8 is a flowchart of the operation of an embodiment of theserver for completion of the qualifying test of FIG. 7.

[0033]FIG. 9 is a flowchart of the operation of an embodiment of theserver for assigning a rating in the coding competition.

[0034]FIG. 10 is a flowchart of the operation of an embodiment of theserver for attracting sponsorship of the coding competition.

[0035]FIG. 11 is a flowchart of the operation of an embodiment of theserver for awarding a prize in the coding competition.

[0036]FIG. 12 is a chart of an embodiment of the progress ofparticipants from elimination rounds of competition to a national roundof a coding competition.

[0037]FIG. 13 is an example screen shot of an embodiment of a web pagedisplayed during the coding competition.

[0038]FIG. 14 is an example screen shot of an embodiment of a codingproblem selection screen displayed during the coding competition.

[0039]FIG. 15 is an example screen shot of an embodiment of a codingwindow displayed during the coding competition.

[0040]FIG. 16 is an example screen shot of an embodiment of a challengewindow displayed during the coding competition.

DETAILED DESCRIPTION

[0041] Referring to FIG. 1, in one embodiment, a coding competitionsystem 100 includes at least one server 104, and at least one client108, 108′, 108″, generally 108. As shown, the coding competition system100 includes three clients 108, 108′, 108″, but this is only forexemplary purposes, and it is intended that there can be any number ofclients 108. The client 108 is preferably implemented as softwarerunning on a personal computer (e.g., a PC with an INTEL processor or anAPPLE MACINTOSH) capable of running such operating systems as theMICROSOFT WINDOWS family of operating systems from Microsoft Corporationof Redmond, Wash., the MACINTOSH operating system from Apple Computer ofCupertino, Calif., and various varieties of Unix, such as SUN SOLARISfrom SUN MICROSYSTEMS, and GNU/Linux from RED HAT, INC. of Durham, N.C.(and others). The client 108 could also be implemented on such hardwareas a smart or dumb terminal, network computer, wireless device,information appliance, workstation, minicomputer, mainframe computer, orother computing device, that is operated as a general purpose computeror a special purpose hardware device solely used for serving as a client108 in the competition system 100.

[0042] Generally, clients 108 are operated by contestants and are usedby contestants to participate in a coding competition. In variousembodiments, the client computer 108 includes either a web browser 116,client software 120, or both. The web browser 116 allows the client 108to request a web page (e.g. from the server 104) with a web pagerequest. An example of a web page is a data file that includes computerexecutable or interpretable information, graphics, sound, text, and/orvideo, that can be displayed, executed, played, processed, streamed,and/or stored and that can contain links, or pointers, to other webpages. In one embodiment, a user of the client 108 manually requests aweb page from the server 104. Alternatively, the client 108automatically makes requests with the web browser 116. Examples ofcommercially available web browser software 116 are INTERNET EXPLORER,offered by Microsoft Corporation of Redmond, Wash., and NETSCAPENAVIGATOR, offered by Netscape Communications Corporation of MountainView, Calif.

[0043] In some embodiments, the client 108 also includes client software120. The client software 120 provides functionality to the client 108that allows a contestant to participate in a coding competition. Theclient software 120 may be implemented in various forms, for example, itmay be in the form of a Java applet that is downloaded to the client 108and runs in conjunction with the web browser 116, or the client software120 may be in the form of a standalone application. implemented in amulti-platform language such as Java or in native processor executablecode. In one embodiment, if executing on the client 108, the clientsoftware 120 opens a network connection to the server 104 over thecommunications network 112 and communicates via that connection to theserver 104. The client software 120 and the web browser 116 may be partof a single client-server interface 124; for example, the clientsoftware can be implemented as a “plug-in” to the web browser 116.

[0044] A communications network 112 connects the client 108 with theserver 104. The communication may take place via any media such asstandard telephone lines, LAN or WAN links (e.g., T1, T3, 56 kb, X.25),broadband connections (ISDN, Frame Relay, ATM), wireless links, and soon. Preferably, the network 112 can carry TCP/IP protocolcommunications, and HTTP/HTTPS requests made by the web browser 116 andthe connection between the client software 120 and the server 104 can becommunicated over such TCP/IP networks. The type of network is not alimitation, however, and any suitable network may be used. Typicalexamples of networks that can serve as the communications network 112include a wireless or wired ethernet-based intranet, a local orwide-area network (LAN or WAN), and/or the global communications networkknown as the Internet, which may accommodate many differentcommunications media and protocols.

[0045] A computer coding contest provider operates a contest server 104,which interacts with clients 108. The server 104 is preferablyimplemented on one or more server class computers that have sufficientmemory, data storage, and processing power and that run a server classoperating system (e.g. SUN Solaris, GNU/Linux, MICROSOFT WINDOWS NT).Other types of system hardware and software than that described herecould also be used, depending on the capacity of the device and thenumber of contestants and the size of the contestant base and thecompetitions. For example, the server 104 may be part of a server farmor server network, which is a logical group of one or more servers. Asanother example, there could be multiple servers 104 that may beassociated or connected with each other, or multiple servers couldoperate independently, but with shared data. As is typical inlarge-scale systems, application software could be implemented incomponents, with different components running on different servercomputers, on the same server, or some combination.

[0046] Referring to FIG. 2, in one embodiment, a server 104 includes aclient communication module 206 that is the interface for communicationwith clients 108. The client communication module 206 can be implementedas software running on one or more servers, or may be implemented as astand-alone server. In some embodiments, the client communication module206 can provide an interface both to client software 120 and to a clientweb browser 116, so that, for example, a web browser 116 can be used bya contestant to access performance information, or to observe theprogress of other contestants, and so on, while the client software 120can be used for participating in a coding competition. The interface toeach of the client software 120 and the client web browser 116 can beimplemented separately or in combination. In other embodiments, theclient communication module 206 can also communicate using otherprotocols or mechanisms.

[0047] The client communication module 206 communicates with theapplication server 212, which provides the main programming logic forthe operation of the system. In one embodiment, the application server212 is implemented as one or more application programs running on aserver class computer, which may be the same or different computer asthe client communication module 206. The application server 212 receivesqueries from contestants via the client communication module 206. Theapplication server 212 also provides the coding competitions. Forexample, the application server 212 tracks the activity in the virtualrooms of a coding competition, and provides the coding problems for thecompetition. As another example, in some embodiments the applicationserver 212 may provide a qualifying test, which is a preliminaryrequirement for participating in a coding competition as a contestant.

[0048] The server 104 also includes a contest server 216. The contestserver 216 handles computationally intensive tasks associated with thecoding competition, for example, the compilation and testing of codesubmitted by contestants in response to a coding problem. In oneembodiment, the contest server 216 also assigns ratings to contestants.Generally, the interface between the application server 212 and thecontest server 216 is designed such that the computationally intensivetasks are queued for processing on the contest server 216, therebyallowing the application server 212 to be responsive, even when loadedby one or more active coding competitions.

[0049] The server 104 also includes a database server 220, which storesdata related to the coding competitions in one or more databases. Forinstance, the database server 220 stores competition information,statistics about the contestants, contestant information, serveravailability, and web traffic information. The database server 220provides data to the application server 212. An example of the databaseserver 220 is an INFORMIX DYNAMIC SERVER, offered by INTERNATIONALBUSINESS MACHINES of Armonk, N.Y.

[0050] Referring to FIG. 3, in one embodiment, the client communicationmodule 206 (FIG. 2) includes an interface server 204, which provides aninterface for communications with the client software 120. In oneembodiment, the client interface server 204 is a software moduleexecuting on a server computer 104. In another embodiment, the clientinterface server is a stand-alone server computer in communication withother components of the server 104, such as the application server 212.In one embodiment, the client interface server 204 communicates with theclient software 120 using a messaging protocol that runs overTransmission Control Protocol/Internet Protocol (TCP/IP) forcommunications between the client software 120 and the client interfaceserver 204. The client interface server 204 is designed to maintaincommunications with the client software 120 of a large number of clients108.

[0051] The client interface server 204 includes an interface connection304. which provides the actual interface to the client software 120. Theinterface connection 304 receives messages from the client software 120and provides them to an interface manager 308 and an interface. serverprocess 312. Likewise, the interface manager receives messages from theinterface server manager 308 and the interface server process 312 andcommunicates those messages to the client software 120.

[0052] The interface server manager 308 manages the allocation of theinterface connection 304 and the interface server process 312 to eachclient software 120. Each time a client 108 (via client software 120)connects to the server 104 (via client interface server 204), individualprocesses are assigned that will exist for the life of the client'sconnection. The interface server manager 308 handles this assignmentprocess. The interface server manager 308 also stores any informationregarding the number and nature of each of the client connections.

[0053] The interface server process 312 is the interface from the clientinterface server 204 to the application server 212. The interface serverprocess 312 communicates with the client software 120 at a higher levelthan the interface connection 304. The interface server process 312 alsocommunicates with the interface server manager 308. For example, theinterface server manager 308 may instruct the interface server process312 to obtain a particular application from the application server 212for use in the coding competition.

[0054] Thus, when a contestant wants to transmit a request to the server104, the client software 120 transmits the request over thecommunications network 112. The client software 120 communicates withthe interface connection 304 to enable communication with the interfaceserver process 312. The interface server process 312 receives therequest from the interface connection 304 and determines what needs tooccur to handle the request. The interface connection 304 alsocommunicates the request to the interface server manager 308 asnecessary to properly manage and allocate resources of the clientinterface server 204 to handle the request.

[0055] The interface server process 312 then transmits the request tothe application server 212. The application server 212 includes businessobjects 316 to process the request. Each business object 316 manages oneor more clients 108. Business objects 316 communicate with the clientsoftware 120 via the interface server process 312. Business objects 316process requests from the client software 120. Examples of theimplementation of a business object 316 include, for instance, aComponent Object Model (COM) object and an Enterprise Java Bean (EJB)object.

[0056] If a request requires data access, the business objects 316 makerequests, via a data access layer 320, to the database server 220. Ifthe requests require processing by the contest server 216, the requestsare passed on to a message queue processor 324. which queues requestsfor operation by the contest server 216, and sends the results back tothe client software 102 via the interface server process 312.

[0057] In some embodiments, the client 108 (FIG. 1) can also communicatewith the server 104 via a web browser. In such an embodiment, thecommunication module 206 (FIG. 2) also includes a web server 208. Theweb server 208 delivers web pages to the client 108 and provides aninterface for communications between the web browser 116 and the server104. Preferably, the web server 208 is an enterprise class web server,such as APACHE from the APACHE FOUNDATION, INTERNET INFORMATION SERVERfrom MICROSOFT CORPORATION, or NETSCAPE ENTERPRISE SERVER from NETSCAPECOMMUNICATIONS CORPORATION, running on one or more server classcomputers, which may be the same or different computers than the clientinterface server 204. The web server 208 may be implemented as anycommunications device that is capable of serving web content to clientweb browsers 116 over the communications network 112, and communicatingwith the other parts of the server 104, such as a personal computer, webappliance and so on. The web server 208 and the client interface server204 are each in communication with the application server 212. The webserver 208 provides the interface for web browsers 116 to communicatewith the server 104. The web server 208 includes an HTTP listener 328that receives HTTP requests from the web browser 116 and transmits HTTPreplies to the browser 116. The HTTP listener 328 communicates with theapplication server 212 via a servlet engine 332.

[0058] The application server 212 communicates with web browsers 116 viathe servlet engine 332. The servlet engine 332 uses Java servlets togenerate web pages in response to web page requests. An XSL processor336 interprets XSL style documents, which are used by the servlet engine332 to produce HTML documents. Data requests are made by the servletengine 332 to a business object 316, which makes data requests, ifnecessary, via the data access layer 320 to the database server 220. Theservlet engine 332 combines XML objects and XSL style files to produceHTML files, which are then communicated, via the web server 208, to theclient's browser 116.

[0059] The database server 220 includes one or more databases 340. Thedata access layer 320 communicates with the database 340 to retrievedata. The database 340 can be implemented using commercially availabledatabase software, with standard interfaces. The various types ofinformation in the databases 340 can be stored in the same database orin different databases.

[0060] In one embodiment, the contest server 216 includes a codecompilation engine 344. The code compilation engine 344 compilescontestants' computer code. Here, the term compilation is used in itsbroadest sense, and includes more than just translation of source codeto object code. For example, if the competition system is used forhardware design, the compiler can include tools to translate code into acircuit design format suitable for simulation. The code compilationengine 344 can include any compiler. The choice of code compilationengine 344, however, affects the coding competition such that thecompiler should supports the competition coding languages and testingenvironment. Thus, in one embodiment, the server 104 may have the codecompilation engine 344 compile C or C++ code (e.g., a C++ compiler)usingthe GNU C compiler (GCC) from the FREE SOFTWARE FOUNDATION, Cambridge,Mass. In one embodiment, the server 104 has the code compilation engine344 compile JAVA code using a compiler available from SUN MICROSYSTEMSof Mountain View, Calif. In another embodiment JAVA code is interpretedwithout compilation. In some embodiments, the code compilation engine344 includes or has access to compilers for the various coding languagesor formats to be used in the coding competition.

[0061] In one embodiment, the contest server 216 also includes a codetesting engine 352. which tests the results of contestants' computercode against test data, such as a predetermined test suite. The testinput may come from various sources. Generally the contest server 216runs, interprets, or simulates the test environment, such that the testdata can be provided as input. and the results obtained. Preferably, thecompilers used by the compilation engine 344 compile code such that thecode can be executed by the code testing engine 352. In one embodiment,the contest server 216 additionally includes a rating engine 348 tocompute and manage the contestants' ratings. The rating engine 348 mayimplement any algorithm, such as an algorithm based on statistics, thataccurately represents a contestant's skill level relative to othercontestants. Each of these tasks, code compilation, code testing, andrating generation, are computation intensive. Requests for theseoperations are queued by the message queue processor 324, and providedto the engines 344-352 in order of receipt. This allows the businessobjects 316 to complete the requests without blocking or waiting for theengines 344-352.

[0062] The de-coupling of these computation-intensive tasks in thecontest server 216 from the operation of the other servers allows forscalability of the competition system. Any combination of the servers,processors, or engines (e.g., web server 208, application server 212,message queue processor 324, rating engine 348) can be implemented onthe same or independent processors or computers relative to the otherservers, processors, or engines. Several instances of the contest server216 can execute in parallel (on the same or different computers) torespond to requests from the message queue processor 324. Additionalprocessing capacity can be added quickly to handle these tasks morequickly if the system comes under heavy load, as could happen during acompetition. Thus, a useful feature of this embodiment of the server 104is that it can reliably support a computation-intensive codingcompetition with large numbers of contestants.

[0063] Referring to FIG. 4, in one embodiment, the interaction ofcontestants with the server 104 (via the client 108) can be summarizedgenerally by four steps. The contestant optionally registers with theserver 104 (STEP 404). Preferably, the contestant provides informationabout herself in the registration process. Examples of contestantinformation can include such information as the contestant's name,contact information (i.e., mail address, telephone number, emailaddress), a username or “alias” for use on the site, work experience,education background, certain demographic information, specifictechnical skills, whether the contestant is currently seeking a job,and/or geographic preferences of the contestant for employment. Thealias is how the contestant will be known on the system to othercontestants and spectators. The registration may also include. forinstance, each contestant completing a qualifying test, or qualifyinground, as discussed in more detail below.

[0064] After registering, the contestant can participate in acompetition. In one embodiment, competitions are ongoing on the server104 and held continuously. In other embodiments, competitions are heldat set times, or when announced by the competition administrator. Tocapture some coders' attention and participation, for example, it may bebetter to hold competitions periodically after normal business hours. Inone embodiment, contestants enter a competition at any time. Forexample, a contestant may connect to the server 104 and may enter acompetition at that time if there are other suitable contestantsavailable to compete. In one embodiment, this type of competition isreferred to as a “pick-up” competition because a contestant can enterthe competition at any time that other contestants are available.Competitions may be scheduled such that the server 104 automaticallybegins the competition at the set time (or some predetermined delayafter the predetermined time).

[0065] A competition first involves communication of the coding problem(STEP 406). The server 104 transmits one or more computer codingproblems to each contestant via the client 108 in the competition. Thecontestant reviews the one or more coding problems and develops computercode as a solution to the coding problem (STEP 408). The contestant mayselect one or more of the transmitted coding problems to solve. Thecontestant may select a first problem to solve, and then select anotherproblem to solve. For simplicity, one coding problem may be described,but it is intended that the principles can be extended to more than one.

[0066] In one embodiment, before a contestant submits computer code as afinal answer to the coding problem, the contestant can submit the codefor compilation by the server 104, and for execution with test dataprovided by the contestant. In this way, the contestant can compileand/or run the computer code to determine, for example, how well thecomputer code operates, the speed at which it operates, and the accuracyof the computer code. The contestants can additionally view, forexample, the result of compilation and the results of the execution withthe test input submitted by the contestant.

[0067] In one embodiment, the server 104 records the elapsed time takenby the contestant in the coding stage. The server 104 starts a codingtimer upon transmission of the computer problem to the contestants (viathe clients 108) and ends the coding stage when contestant submits asolution, or when the coding timer reaches a predetermined coding stagetime (i.e. a time-out). Moreover, the server 104 may adjust thepredetermined coding time depending on the level of difficulty of thecomputer problem or problems.

[0068] In some embodiments, after submission of code, a challengeportion of the competition follows (STEP 412). During a challengeperiod, a challenging party can challenge a contestant's computercoding. In various embodiments, the challenging party can be one or moreof: another contestant, a spectator of the coding competition, a judgeor other party who has the assigned task of submitting test data toverify code submissions, or the server 104. In one embodiment, thechallenging party challenges the computer code by submitting test datato the server 104 for input to the executed computer code. The server104 uses the test data to determine the correctness of the computercode. With the test data, the challenging party attempts to induceincorrect execution of the computer code, for example, generating anincorrect output, or halting execution due to error. Based on thecomputer code's response to the test data, the server 104 can determineif the computer code execution is acceptable. In some embodiments, theserver 104 ends the challenge when a challenge timer reaches apredetermined challenge time. Although described as independent timers,the timers such as the challenge timer and the coding timer, may beimplemented with a single timer or multiple timers.

[0069] The server 104 may also conduct automated testing of thesubmitted code (STEP 416) in which the code is submitted to one or moreautomated tests from a predetermined test suite. In one embodiment, theserver 104 submits all submitted computer code to one or more automatedtests. The server 104 may randomly choose a test from the predeterminedtest suite or may choose a test based on, for example, a predeterminedorder or the computer code's performance with respect to an earlierchallenge. In another embodiment, the challenge stage includes automatedtesting. In another embodiment, the automated testing precedes thechallenge stage.

[0070] In one embodiment, a competition includes a contestantregistering and then participating in two or more competition rounds,where each round consists of problem communication (STEP 406), coding(STEP 408), challenge (STEP 412), and testing (STEP 416). Alternatively,a competition round may include a subset of problem communication (STEP406), challenge (STEP 412), and testing (STEP 416). In some embodiments,the order of challenge (STEP 412) and automated testing (STEP 416) isreversed and in other embodiments, these steps overlap or take place inparallel. In one embodiment, a coding competition consists of apredetermined number of rounds, where the number varies from competitionto competition. Thus, a competition may only consist of one round. Acompetition with a number of rounds may also be referred to as atournament. As discussed below, the server 104 can additionally assignor adjust each contestant's rating after the entire competition iscomplete (e.g., after the round(s) of the competition are complete). Around can include communication of two or more problems, withoverlapping problem communication, challenge, and automated testing.

[0071] Referring to FIG. 5, in one embodiment, the server receivescontestant information from the client (if it is the first competitionfor the contestant) or the database (STEP 504). In some embodiments. theserver 104 does not enable the contestant to move to the next step untilrequired contestant information is received by the server 104. In someembodiments, the server 104 requests additional contestant informationfrom the client 108 at an appropriate time, such as during anintermission of a coding competition. In other embodiments, if acontestant has registered with the server 104 but has never competed ina competition, the contestant may not have to complete additionalinformation before competing in a competition. Thus, in one embodiment,all contestants provide a minimum of registration information prior tocompeting.

[0072] In another embodiment, the server 104 hosts a practice room. Thepractice room is a room in which one or more contestants can familiarizethemselves with the competition format (e.g., the format for answeringcoding problems). In one embodiment, inside the practice room, theserver 104 provides sample coding problems to one or more contestants.The contestant(s) attempt to develop code in response to these problems,which may help familiarize them with the competition system.

[0073] In one embodiment, the server 104 determines if enoughcontestants are available to begin the coding competition (STEP 508). Ifthere are not, the server 104 waits until a predetermined number ofcontestants connect (via a client 108) to the server 104 forparticipation in a coding competition. In some embodiments, the server104 may wait a predetermined time after establishing connections withtwo contestants (via clients 108) and then begin a competitionthereafter. For instance, if the server 104 schedules a codingcompetition to begin on a particular date and time, the server 104 mayonly wait a predetermined amount of time (e.g., ten minutes) past thescheduled time before beginning the competition, as long as more thanone contestant is going to participate in the competition. If nocontestants or only one contestant submits contestant information at thescheduled time, the server 104 may automatically cancel or postpone thecoding competition to a later date or time.

[0074] When a predetermined number of clients 108 associated withregistered, qualified contestants are available to participate, theserver 104 transmits a coding problem to each contestant via the client108 (STEP 512). In one embodiment, the coding problem requires acontestant to create one or more computer programs, subroutines, and/ortasks. In one embodiment, the coding problem involves developing code inone or more specified languages or formats.

[0075] The server 104 can also impose a time limit on its acceptance ofresponses to the coding problem. In one embodiment, the server 104displays a response timer on the client 108 when the server 104transmits the coding problem. The server 104 sets the response timer toa predetermined response time to denote the time limit in which theserver 104 accepts computer code in response to the coding problem. Whenthe response timer expires, the contestant can no longer submit computercode.

[0076] In some embodiments, a contestant chooses one coding problem fromseveral transmitted coding problems. The server 104 transmits severalcoding problems to the client 108, where each coding problem has adifferent level of difficulty with respect to other coding problems. Ifa contestant submits a correct solution to a coding problem having ahigher level of difficulty, the more difficult choice can result in anawarding of more points relative to a correct solution to a codingproblem having a lower level of difficulty. After the contestant submitscode in response to one problem, the contestant may be able to work onanother coding problem. In another embodiment, the coding problemsoffered in a competition have the same degree of difficulty andtherefore have the same ascribed point value. In another embodiment,there is only one problem that must be responded to, with no choice.

[0077] The contestant may compile (if necessary) and test her codebefore submitting the code to the server 104 as a final response. Thecontestant can thus validate her code before submission. By compilingand/or testing the computer code, the contestant may be able to, forexample, improve the computer code, determine errors in the computercode, determine the accuracy of the computer code, and determine theefficiency of the computer code. When the code is ready, the contestantthen submits her computer code to the server 104. The server 104receives the computer code from a contestant (STEP 516). The server 104compiles the code (STEP 520) if necessary. If the server 104 receivesexecutable or interpretable computer code (e.g. a JAVA program) from acontestant, there may be no need for a compilation step.

[0078] In one embodiment, the server 104 waits a predetermined amount oftime to receive computer code from each contestant. Once thispredetermined coding time has elapsed, the server 104 does not acceptcode submissions. In other embodiments, the server 104 continues toaccept code submissions, but the points available for the submissiondecrease over time.

[0079] The server 104 verifies the correct execution of the submittedcomputer code (e.g., does it generate correct output. execute withouterrors in all cases, etc.). The server does this by obtaining test data(STEP 524), which may come from various sources. In one embodiment, theserver 104 has predetermined test data to provide as input to thesubmitted coding problems. Such test data is provided to the server withthe coding problem. In another embodiment, the server can generate testdata using information about the types of input that can be provided tothe computer code (STEP 524). As described above, test data may also beprovided as a challenge by other contestants, or in some embodiments, byspectators or judges (FIG. 4, STEP 412). The server 104 may choose thecontestant's computer code to verify based on one or more factors, forexample, the order of submission of computer code to the server 104, theorder of test data provided to the server 104, the current ranking ofthe submitting contestant, the current amount of points awarded to thecontestant in this competition, as described below, randomly, or in apredetermined order. In one embodiment, in order to enable thisautomated verification of computer code responses, when test questionsare provided to the server 104 for use in a coding competition, areference computer code 636 as well as predetermined test data areprovided to the server 104 along with the test question. The referenceprogram 636 and the test data are stored with the test question for usein this manner.

[0080] The server 104 executes the code with the test data as input anddetermines the computer code's response (STEP 528). In one embodiment,the server 104 can display the results of the execution of the computercode with the test data as input to contestants and spectators. Inanother embodiment, the test data used and the results obtained arestored in a database for later viewing and verification.

[0081] In one embodiment, the server 104 generates test data andsubjects the code to the server's data as part of the automated testingstage. Even if the server 104 previously provided or generated test dataas a challenge, in one embodiment the server 104 inputs one or moreadditional test datum to the computer code.

[0082] The server 104 evaluates the response of the computer code (STEP532). In one embodiment, the server 104 evaluates the response bycomparing the response to an acceptable response or a range ofacceptable responses. Further, if the contestant's computer code failsor produces errors when executing with the test data as input, theserver 104 may deduct points from the submitting contestant. Moreover,if the computer code properly executes with the test data, the server104 may award points to the submitting contestant. In some embodiments,the server 104 may also add or deduct points from a contestant whosubmits challenge test data for a particular computer code if thechallenged computer code executes correctly in response to the testdata. In other words, the system will give points for a successfulchallenge and deduct points for an unsuccessful challenge of acontestant's computer code.

[0083] In one embodiment, in which the communications network 112 is theInternet, competitions take place in a “virtual forum” setting, in whichspectators can observe, via the computer network, the activity in avirtual competition room. Spectators may be users of a client 108 thathave registered with the server 104 who are not participating in aparticular coding competition, or may be observers who have not yetregistered with the server 104. In some such embodiments, besides theserver 104 and other contestants competing in the coding competition,spectators (either locally or via the network) can participate in thecompetition by submitting test data to the server 104 for input into thecontestants' submissions. Thus, spectators can help judge the worthinessof the submissions.

[0084] Also referring to FIG. 6, in one embodiment, the server 104 usesreference code 636 to verify the operation of a contestant's computercode 640. Preferably, the reference program 636 is a predeterminedsolution to a coding problem. Test data 639 is provided as input both tothe reference program 636 and the contestant's code 640. The test datamay be predetermined data stored by the server 104, data that isgenerated by the server 104, data provided by another contestant, ortest data provided by another party, such as a spectator, judge,administrator, etc. The reference program 636 produces a referenceresponse 638 during (or following) execution with the test data asinput. The contestants' computer code 640 also produces a response 644when it is executed with the test data as input. The server 104 thencompares the two responses 638, 644 to determine the correctness of thecomputer code response 644. The server 104 may also measure the timethat the reference program 636 and the contestant's computer code 640take to produce an associated response 638, 644 to determine theefficiency of the contestant's computer code 640. This comparison may berepeated for various input test data, with each test input coming fromthe same or a different source.

[0085] Referring again to FIG. 5, following the evaluation of thecontestant's computer code, the server 104 awards points to thecontestant (STEP 536). In one embodiment, each coding problem has amaximum point value associated with it. In one embodiment, the server104 awards less points than the maximum point value that the codingproblem is worth. In particular, the server 104 awards points to acontestant based on, for example, some or all of the correctness of thecontestant's code response 644 (e.g. compared with the referenceresponse 638), the accuracy of the contestant's computer code response644, the speed that the computer code 640 produces the computer coderesponse 644, and/or the recorded time at which the server 104 receivedthe computer code 640 from the contestant.

[0086] In one embodiment, the number of points awarded depends on thelevel of difficulty of the coding problem. The server 104 may alsoadjust the point value awarded to a contestant based on the time thatthe server 104 receives the contestant's computer code 640 compared tothe time that the server 104 provides the coding problem. In oneembodiment, the total number of points awarded to a contestant is asshown in Equation 1:

Total points awarded=MP*((0.3+(0.7*TT ²))/(10*PT ² +TT ²))  Equation 1

[0087] In Equation 1, PT is the time a contestant spends coding aproblem (i.e., producing the computer code), TT is the total timeallocated for coding all computer problems in this part of thecompetition, and MP is the maximum points available for the computercoding problem. In one embodiment, the maximum number of pointsavailable is a frame of reference for the server 104 that is notpractically achievable by a contestant. More specifically, for acontestant to be awarded the maximum number of points, a contestantwould have to spend almost zero seconds reading the coding problem,coding the computer code 640, and submitting the computer code 640 tothe server 104. For example, if there is one problem that is allocated600 seconds (10 min.), and it takes a contestant 300 seconds (5 min.) tocode the problem, then TT=600, PT=300, and approximately 52% of thetotal points MP will be awarded if the code executes correctly.

[0088] In one embodiment, the contestant receives this number of points(e.g. 52% of the total available) for the computer code if the computercode executes correctly with all test data, and the contestant receivesno points if the code does not execute correctly with all test data. Inother embodiments, partial credit is given for partially correct code.For example. in one embodiment, the code receives the number of pointsbased on a percentage of test data. In another embodiment, the number ofpoints received is scaled based on the percentage of test data withwhich the code executes correctly. In some embodiments, such as when thecompetition is a tournament consisting of multiple rounds, the server104 tracks the points awarded to each contestant for each round. Theserver 104 may also calculate a running total for each contestant.summing the points awarded in each completed round of the competition.This running total can be used to eliminate low-scoring contestants.

[0089] In one embodiment, after the completion of one or morecompetition rounds, the server 104 assigns a rating to the contestant(STEP 540). Typically the assignment of the rating occurs when acompetition or tournament is complete. For example, a competition mayconsist of one round, as described above with respect to FIG. 4, or mayconsist of multiple rounds, such as in a tournament. Thus, in theembodiment in which the competition consists of one iteration of theproblem communication, coding, challenge, and testing steps, or morethan one round, the server 104 assigns a rating to each contestant afterawarding total points to each contestant. In one embodiment in which thecompetition includes multiple rounds (e.g., a tournament), the server104 assigns a rating to each contestant after the contestants completethe tournament.

[0090] The ratings provide a metric that can be used to chart aprogrammer's skill, knowledge, and ability. As described in more detailbelow, in one embodiment the server 104 bases the contestant's rating onthe total number of points awarded to the contestant during thecompetition (e.g., having one or multiple rounds). The rating can be ascore, a grade, or any other type of rating technique.

[0091] Referring to FIG. 7, in one embodiment, a contestant undergoes aqualifying test as part of the registration process. The qualifying testcan take place at the time of registration, or at a later time. Duringthe qualifying test, the server 104 transmits one or more qualifyingtest questions to a contestant (STEP 704). In one embodiment, thecontestant has to complete the qualifying test in a predeterminedqualification time, which the server 104 displays on the client 108. Inone embodiment, the predetermined qualification time is a time for theentire qualification test. Alternatively, the predeterminedqualification time is a time requirement for one qualifying testquestion. In one embodiment, if the predetermined qualification timeelapses, the server 104 transmits the next question in the qualifyingtest. In another embodiment, the server 104 terminates the qualifyingtest. The server 104 may also time both the entire qualifying test andan answer to each qualifying test question.

[0092] The questions may be, for example, multiple choice questions,questions requiring one word answers, or a question that requires thatthe contestant write computer code. Preferably, the questions arecomputer-related and correct response indicates a base level of skill.The questions may, however, be related to any subject, such as, forinstance, mathematics, science, literature. history, or logic. Moreover,in one embodiment, the qualifying test questions have varying degrees ofdifficulty. Further, the server 104 may or may not display these degreesof difficulty to the contestant, and the contestant may, before viewingthe qualifying test question, have a choice of which qualifying testquestion to answer based on its level of difficulty.

[0093] The contestant (via the client 108) answers each qualifying testquestion and then submits the answer to the server 104. The server 104receives the answer (STEP 708) from the client 108. In one embodiment,the server 104 evaluates the received answer (STEP 712). In anotherembodiment, once the contestant answers the test question, the client108 (e.g., the web browser 116 or the client software 120) automaticallytransmits the answer to the server 104 and the server 104 provides theclient 108 with the next qualifying test question (STEP 704).

[0094] During the evaluation of the received answer, the server 104 may,for instance, determine the value of the received answer based on apredetermined set of correct answers and the speed at which thecontestant responded to the qualifying test question. The server 104 mayadditionally determine a number of points to award to the contestant forthe contestant's answer to the qualifying test question, or qualifyingtest question score, based on the previous determinations (e.g.,correctness of the received answer). After this evaluation, in oneembodiment, the server 104 notifies the contestant about the result(s)of the evaluation of the contestant's answer, such as, for example, thecorrectness of the contestant's answer or the qualifying test questionscore. In one embodiment, the server 104 notifies the contestant via aqualification results window displayed on the client 108 as thequalification test progresses. In another embodiment, the server 104only notifies the contestant about the result(s) of the evaluation ofthe qualifying test at its completion.

[0095] The server 104 determines if the qualifying test is complete(STEP 720). The server 104 establishes that the qualifying test iscomplete based on, for example, when the predetermined qualificationtime for the qualifying test elapses, when the server 104 receives ananswer to the final qualifying test question, or when a predeterminedtime for the final qualifying test question elapses. If the server 104determines that the qualifying test is not complete after the receptionof the contestant's answer to the current qualifying test question, theserver 104 subsequently transmits the next qualifying test question inSTEP 704. If, however, the server 104 determines that the qualifyingtest is complete after the reception of the contestant's answer to thecurrent qualifying test question, the server 104 determines if thiscontestant qualifies for a coding competition.

[0096] Thus, after determining that the qualifying test is complete, theserver 104 may then evaluate the answers submitted by the contestant(STEP 724). In this embodiment, the server 104 can then notify thecontestant of the results of the evaluation (STEP 728). Thisnotification may be a notification as to the correctness of eachquestion, the time taken to answer each question, the time taken tocomplete the qualifying test, the level of difficulty of each testquestion, the qualifying test question score, total points awarded forthe qualifying test, or qualifying test score, and/or a minimum numberof points needed to qualify for the coding competition. The pointsawarded per question may depend on if the answer was correct, the timetaken to receive the submitted answer, and the level of difficulty ofthe question. In one embodiment, participants have the option of takingthe qualifying test a second time should they want to try to improvetheir score to increase their chances of qualifying.

[0097] In one embodiment, after determining that the qualifying test iscomplete, the server 104 allows the user to enter a coding competition.In this embodiment, the qualification test is an opportunity for a newcontestant to become familiar with the competition environment andpractice before a coding competition begins.

[0098] Referring to FIG. 8, in determining whether a contestant canparticipate in a coding competition, the server 104 looks at thequalifying test score of the contestant, and the number of contestantsalready participating.

[0099] The server 104 first determines if the qualifying test score isgreater than a predetermined threshold qualifying test score (STEP 804).The contestant can not participate in the coding competition (STEP 808).

[0100] If the qualifying test score is greater than the predeterminedqualifying test score, the server 104 then increments a contestantcounter, which represents the number of contestants who have qualifiedfor the coding competition (STEP 812). If, however, the server 104determines that the contestant counter is less than or equal to thepredetermined maximum contestant count (STEP 816), the contestant isallowed to compete (STEP 824).

[0101] In one embodiment, the registration process includes the server104 seeding each contestant into a division. In one such embodiment, theserver 104 allocates two divisions, a first division for contestants whohave previously attained a rating greater than a predetermined divisionrating and a second division for contestants who have either notattained a rating (i.e., who have never competed before or have no dataassociated with a competition) or have a rating below the predetermineddivision rating. In one embodiment, the server 104 provides codingproblems having a lower degree of difficulty to the second divisionrelative to the problems provided to the first division.

[0102] For example, the server 104 could use a predetermined divisionrating value of 1200. Thus, the server 104 seeds all contestants whohave a rating less than 1200 into the second division and allcontestants who have a rating greater than 1200 into the first division.Moreover, in one embodiment, the server 104 assigns all contestantsagain after the completion of a competition. Thus, after a competition,if a contestant previously in the first division has a rating below 1200after this competition, the server 104 places the contestant in thesecond division. Further, the server 104 can select the prize that thecontestants are competing for depending on the division that thecontestants are competing in.

[0103] The server 104 may further divide up each division into one ormore “rooms” or group of rooms. For example, the server 104 can divideup the second division into unrated rooms having only contestants whohave never been assigned a rating, and rated rooms having contestantswho have been assigned a rating below the predetermined division rating.

[0104] Referring to FIG. 9, the server 104 quantifies a contestant'scoding abilities after the completion of some or all of a competition byrating the contestant. In one embodiment, the server 104 determines therating based on the total points awarded to the contestant in thecompetition.

[0105] As above, the server 104 electronically communicates a codingproblem to each contestant (STEP 904). The server 104 then receives acontestant's computer code in response to the problem (STEP 908). Theserver 104 then evaluates the received code (STEP 912).

[0106] Based on these evaluations of the code, the server 104 thenawards points to the contestant (STEP 916). Once some or all of thecompetition is complete (e.g., one round or multiple rounds), the server104 determines each contestant's rating (STEP 920). In one embodiment,to determine a contestant's rating, the server 104 first determines acontestant's previous rating if the contestant has been ratedpreviously. Preferably, the server 104 retrieves the ratings of thecontestants from, for instance, the database server 220. The ratingincludes a rating value and a volatility value. The volatility valueindicates the consistency of the contestant's ranking over time.

[0107] In one embodiment, the server 104 calculates the average ratingfor the contestants participating in a competition using Equation 2:$\begin{matrix}{{AveRating} = \frac{\sum\limits_{i = 1}^{NumCoders}{Ratings}_{i}}{NumCoders}} & {{Equation}\quad 2}\end{matrix}$

[0108] The server 104 also calculates a competition factor usingEquation 3: $\begin{matrix}{{CF} = \sqrt{\frac{\sum\limits_{i = 1}^{NumCoders}{Volatility}_{i}^{2}}{NumCoders} + \frac{\sum\limits_{i = 1}^{NumCoders}( {{Rating}_{i} - {AveRating}} )^{2}}{{NumCoders} - 1}}} & {{Equation}\quad 3}\end{matrix}$

[0109] In Equations 2 and 3, NumCoders is the number of contestants inthe competition and Rating_(i) is the rating without the volatility ofthe ith contestant in the competition before the competition. Further,Volatility_(i) is the volatility of the ith contestant in thecompetition before the competition.

[0110] Next, the server 104 determines the skill of the contestant, asshown below in Equation 4. $\begin{matrix}{{Skill} = \frac{{Rating} - 1200}{1200}} & {{Equation}\quad 4}\end{matrix}$

[0111] The Rating is the contestant's rating before the competition, ora previous rating. In one embodiment, if the server 104 determines thatthe contestant has not been previously rated, the server 104 does notcalculate the skill. In a further embodiment, the server 104 assigns apredetermined skill level to a contestant (e.g. 1). The servercalculates the deviation of a contestant from other contestants usingEquation 5, where Volatility is the volatility of the contestant beforethe competition: $\begin{matrix}{{Deviation} = \frac{Volatility}{1200}} & {{Equation}\quad 5}\end{matrix}$

[0112] The server 104 then estimates the probability (WP_(i) for i from1 to NumCoders) of the server 104 awarding a contestant a higher scorerelative to the other contestants in the competition. In one embodiment,the server 104 estimates this probability by using the contestant'sexpected performance rating on a range of good to bad days. The server104 models each contestant's performance as a normal distribution with amean (the rating) and standard deviation (the volatility). Using thismodel, the server 104 converts a contestant's performance into apercentage using the normal distribution, so that 50% representsexpected performance and 25% represents better three days out of four,etc. Then, to estimate the probability that the contestant beatscontestant i, the server 104 divides the range from 0% to 100% into 1%blocks, and the server 104 takes the midpoint of each block and computesthe corresponding performance. The win probability is the number ofblocks where the midpoint favors contestant i. If two contestants havethe same number of blocks in which the midpoint favors one contestant(i.e., a tie), the server 104 splits the block between both contestants.

[0113] In particular, the psuedo-code function to estimate theprobability of contestant A beating contestant B, where f is the inverseof the standard normal function, is shown below:

[0114] WinProbability(SkillA, DeviationA, SkillB, DeviationB)

[0115] Set count, a, b to 0. Set fa=SkillA+f((a+0.5)/100)*DeviationA.

[0116] Set fb=SkillB+f((b+0.5)/100)*DeviationB.

[0117] While (a<100):

[0118] If fa<fb:

[0119] Add 100-b to count, Increment a,

[0120] Set fa=SkillA+f((a+0.5)/100)*DeviationA,

[0121] If fa==fb:

[0122] Add 100-b-0.5 to count, Increment a, Increment b,

[0123] Set fa=SkillA+f((a+0.5)/100)*DeviationA,

[0124] Set fb=SkillB+f((b+0.5)/100)*DeviationB

[0125] If fa>fb:

[0126] Increment b

[0127] Set fb=SkillB+f((b+0.5)/100)*DeviationB

[0128] Return count, where count is the win probability.

[0129] The server 104 then calculates the expected rank of thecontestant using Equation 6: $\begin{matrix}{{ERank} = {\sum\limits_{i = 1}^{NumCoders}{WP}_{i}}} & {{Equation}\quad 6}\end{matrix}$

[0130] Where Wpi is the Win Probability of a coder against each othercoder. The server 104 also calculates the expected performance of thecontestant using Equation 7: $\begin{matrix}{{EPerf} = {- {\Phi ( \frac{{ERank} - {.5}}{NumCoders} )}}} & {{Equation}\quad 7}\end{matrix}$

[0131] In Equation 7, Φ is the inverse of the standard normal function.The server 104 then determines the actual performance of eachcontestant: $\begin{matrix}{{APerf} = {- {\Phi ( \frac{{ARank} - {.5}}{NumCoders} )}}} & {{Equation}\quad 8}\end{matrix}$

[0132] In Equation 8, ARank is the actual rank of the contestant in thecompetition based on score (1 for first place, NumCoders for lastplace). If the contestant's score was equal to another contestant, therank is the average of the positions covered by the tied contestants.The server 104 then calculates the “performed as” rating of thecontestant using Equation 9:

PerfAs=OldRating+CF*(APerf−EPerf)  Equation 9

[0133] The server 104 also determines the weight of the competition forthe contestant in Equation 10 and a cap for the contestant in Equation11: $\begin{matrix}{{Weight} = {\frac{{TimesPlayed} + 1}{{{.75}*{TimesPlayed}} + {.4}} - 1}} & {{Equation}\quad 10} \\{{Cap} = {150 + \frac{1500}{1 + {TimesPlayed}}}} & {{Equation}\quad 11}\end{matrix}$

[0134] In Equation 10, TimesPlayed is the number of times the contestanthas been rated before. In one embodiment, to stabilize the higher ratedcontestants. the Weight of contestants whose rating is between 2000 and2500 is decreased 10% and the Weight of contestants whose rating is over2500 is decreased 20%. The server 104 then determines the new volatilityof the contestant using Equation 12 and the new rating of the contestantwith Equation 13: $\begin{matrix}{{NewVolatility} = \sqrt{\frac{{Volatility}^{2} + {{Weight}*( {{CF}*( {{APerf} - {EPerf}} )} )^{2}}}{1 + {weight}}}} & {{Equation}\quad 12} \\{{NewRating} = \frac{{Rating} + {{Weight}*{PerfAs}}}{1 + {Weight}}} & {{Equation}\quad 13}\end{matrix}$

[0135] In one embodiment, if |NewRating−Rating|>Cap, the server 104adjusts NewRating so that it is at most Cap different than Rating.Moreover, in some embodiments, the server 104 stores any of the abovementioned values, such as the Rating, Volatility, and Number of timespreviously rated. Additionally, in one embodiment, ratings of newcontestants are provisional before competing in a coding competition.

[0136] Thus, in the embodiment just described, the server 104 does notadjust a contestant's rating drastically from one performance in acoding competition if the contestant has performed and been rated inmany other coding competitions or rounds of coding competitions. Theserver 104 may also determine a maximum rating increase for eachcontestant to help stabilize a contestant's rating from one codingcompetition to another. In one embodiment, the server 104 displays eachcontestant's ratings on the web site of the coding competition. However,in another embodiment the server 104 only enables registered contestantsand sponsors to view the ratings of contestants.

[0137] TABLE 1 illustrates three examples of coding problems transmittedby the server 104 to contestants. As shown in the three examples, codingproblems may include notes to assist the contestant in writing asolution to the problem. Also, coding problems may provide one or moreexamples of correct output of the computer code that solves the problem.TABLE 1 Coding problem Example Value Implement a class Finances, whichcon-  (100, 0, 100) −> 1 250 tains a function timeLeft. TimeLeft is (200, 1, 300) −> 0 passed 3 int values. The initial value (3453, 8,200) −> 1200 of the bank account is initialValue. Interest is compoundedmonthly at a rate of interest percent. Every month monthly dollars istaken out from the account (immediately after the Interest has beenadded). timeLeft must re- turn an int that is the number of months theaccount can support the full monthly withdrawal. If the account willlast longer than 1200 months, the function should return 1200. Themethod signa- ture is: int timeLeft (int initialValue, int interest, intmonthly). Note: initialValue and monthly are > or equal to 0 and < orequal to 100,000. Interest is > or equal to 0 and < or equal to 1000.Implement a class Fraction, which con- If index = 3 and max 500 tains amethod getXSmallest. The denominator—4, all method is passed twointegers: an the fractions of the index and a maximum denominator. forma/b are ¼, ⅓, The program should find all fractions ½, {fraction (2/4)},⅔, ¾, and of the form a/b where b is less than or the method shouldequal to the maximum denominator and return the third a/b is less than 1and then return the smallest, as a double: index-th smallest of thesefractions. 0.5. The method should return the fraction as a double value,rounded to 3 decimal points (use the supplied formatDouble function).Note: If the index value is > the number of fractions that are less than1, the method should return 1. Fractions should not be simplified. ½ and{fraction (2/4)} are two unique fractions, despite their equal doublevalues. The class and method must be declared as public. The index mustbe between 1 and 70, inclusive. The max denomi- nator must be between 2and 12, in- clusive. Implement a class Numbers that con- num = 8: 8 canbe 1000 tains a method pairwisePrimes. written as 1 + 2 + 5 and Themethod inputs an int (num) and 1 + 3 + 4 and the returns the number ofways the number method should return can be written as the sum of threedis- 2. num = 18: the tinct integers that are pairwise rela- methodshould return tively prime; that is, no pair of the 14 three integershas a common factor > 1. Note: num is > 0 and < or equal to 40. One ofthe three distinct integers can be 1; Numbers are Pairwise RelativelyPrime if they share no factor GREATER than 1.

[0138] The coding problems shown in TABLE 1 have different levels ofdifficulty. These levels of difficulty are depicted as the differentpoint values of each coding problem. For instance, a contestant canchoose a 250 point coding problem to earn a maximum of 250 points for acorrect solution. However, another contestant may choose a 1000 pointcoding problem, which has a higher level of difficulty relative to a 250point question, to earn a maximum of 1000 points for a correct solution.Thus, in one embodiment, a contestant having a higher degree of skillcan amass points more rapidly than a contestant having a lower degree ofskill by selecting coding problems with higher point values (andtherefore higher levels of difficulty). In one embodiment, a server 104assigns a higher rating to a contestant that amasses a higher totalpoint value.

[0139] In one exemplary embodiment, multiple coding problems areavailable to all of the contestants in a virtual room. The contestantsare given a choice of which problem to work on. Once a contestant viewsa problem, the timing on that problem begins. When the contestantcompletes the code that is a response to one problem, and submits it tothe server, the contestant can begin work on a second problem, and thenthe third, and so on, depending on the number of problems amount of timethat is allocated to coding in the competition. In some embodiments, thecontestants do not see the problems, but only the point values, and thecontestants select a problem to work on based only on the point value ofthe problem. Thus, each contestant makes a judgement about his abilitiesin deciding which coding problem to work on first.

[0140] Referring to FIG. 10, the server 104 may use the ratings for manypurposes, including for attracting sponsors to the coding competition.The ratings provide a tangible, objective, and quantifiable way forcompanies to determine the skill level of a contestant. As above, theserver 104 performs the registration stage in which the server 104receives contestant information (STEP 1004). The server 104 then runs atleast one coding competition (STEP 1008) and assigns a rating to eachcontestant (STEP 1012).

[0141] The competition administrator can then use these competitions toattract sponsorship. In particular, the server 104 provides contestantinformation about one or more contestants to a first sponsor in exchangefor sponsorship (STEP 1016). Sponsorship can include many things, butgenerally includes providing financial support for the competitionand/or for the competition administrator. For example, in oneembodiment, a sponsor provides the prize money that is awarded tocontestants who have scored a higher number of point values relative toa predetermined number of other contestants. Although described below assponsoring the entire coding competition, a company can also sponsor apart of the coding competition (e.g., a round of a coding competition).In one embodiment, the server 104 causes the client to display a logoand other promotional information associated with a sponsor of thecoding competition on the contestant's web browser 116 (FIG. 1) orclient software 120 (FIG. 1). Moreover, the server 104 can provideselected contestant information and ratings to a sponsor.

[0142] In one embodiment, the competition administrator tailors theinformation about contestants that the server 104 provides to thesponsor. In this embodiment, the server 104 transmits only portions ofcontestant information (STEP 1020). For example, the criteria may be ageographic preference for employment, years of work experience, codinglanguage specialty or other specific skills, and current employmentstatus of the contestant (e.g., active job hunting, passive jobhunting).

[0143] The criteria may also be a rating range that a sponsor willconsider for employment opportunities. If the contestant is rated lowerthan the minimum rating for that sponsor, the server 104 does notprovide that contestant's information to the sponsor. In otherembodiments, the server 104 receives the criteria from a sponsor at thetime the company decides to sponsor the coding competition. Thus,although the reception of the criteria is shown in STEP 1020 after theserver 104 provides the contestant information to the sponsor, theserver 104 may alternatively receive the criteria before thetransmission of information to the sponsor in STEP 1016.

[0144] After the server 104 transmits contestant information forparticular contestants to a sponsor, the sponsor may then contact thecontestant for employment opportunities. In one embodiment, the sponsorcommunicates with the competition administrator to obtain contestantinformation for potential employment. Further, the sponsor communicateswith the competition administrator and the competition administratorthen communicates with the contestant regarding requested informationand employment opportunities. Thus, the competition administrator actsas an agent for the contestants to find employment.

[0145] In one embodiment, the server 104 provides information to thesponsor about one or more contestants and the sponsor subsequentlycontacts the contestant. In one embodiment, the sponsor contacts thecontestant through a communication service (e.g. electronic messaging)provided by the server 104. In this way, the actual identity of thecontestant can be hidden from the prospective employer, and thecontestant can decide whether to respond to the inquiries made by thesponsor. Additionally, a contestant who has a high rating relative toother contestants may use that rating to bolster the contestant'snegotiating power when negotiating a starting salary and/or benefitswith the prospective employer.

[0146] In one embodiment, the competition administrator provides one ormore advertisements to each contestant for viewing during thecompetition. The advertisements may be displayed on the web browser, theclient software, or some combination. One or more advertisers supplythese advertisements to the competition administrator. An advertiser maybe a sponsor of the coding competition and may further be a company orfirm that hires coders, develops software and/or hardware, and/or isinvolved with technology. The advertiser may also be the competitionadministrator. In one embodiment, the server 104 displays one or moreadvertisements on the client software 120. An advertisement may include,without limitation, a logo, a quote, a word, a paragraph, a picture, ora sound or video clip. The server 104 then tracks the viewing time ofthe advertisements by each contestant. For instance, the server 104 candetermines the amount of time that the client software displayed aparticular advertisement. The server 104 then provides contestantinformation about the contestants viewing the advertisement as well asthe tracked viewing time of the advertisement to the advertiser.

[0147] In one embodiment, the server 104 provides the same advertisementto each contestant. The server 104 then provides contestant informationto the advertiser in the aggregate. The contestant information mayinclude, for instance, the experience of the contestants viewing theadvertisement, the rating of each contestant, and/or the geographicpreference of each contestant. For example, the server 104 may providethat ten extremely skilled contestants having a rating above or equal to2000 viewed the advertisement for a total time of 30 minutes, twelvehighly skilled programmers having a rating above or equal to 1800 butbelow 2000 viewed the advertisement for 27 minutes, and twenty skilledprogrammers having a rating above or equal to 1500 but below 1800 viewedthe advertisement for 31 minutes. The server 104 may also provide a listof the time each contestant viewed the advertisement. In someembodiments, the server 104 provides other information about each of theabove-mentioned contestants, such as that they have a geographicpreference of Chicago, Ill.

[0148] In another embodiment, the server 104 provides differentadvertisements to different contestants based on contestant information.For example, if an advertiser is a computer organization located only inNew York and develops software using the Java programming language, theserver 104 displays this advertiser's advertisements to contestantshaving contestant information matching the above information (i.e.,geographic preference of New York, knowledgeable in the Java programminglanguage). In one embodiment, the advertiser specifies to thecompetition administrator (or server 104) that the server 104 shouldonly provide their advertisement to coders having particular informationas part of their contestant information. In another embodiment, theserver 104 increases the frequency of viewing opportunities of certainadvertisements based on information that the server 104 stores relatedto the particular advertiser.

[0149] Referring to FIG. 11, in addition to access to employmentopportunities, a contestant who earned a large number of points in oneor more rounds of a tournament relative to other contestants may beawarded one or more prizes. As above, the server 104 provides the codingcompetition (STEP 1104) and awards points to each contestant during(e.g., points earned after each round) and following (e.g., totalpoints) the competition (STEP 1108). The server 104 determines if acontestant scoring a particular number of points should receive a prize.The server 104 makes this determination based on whether the awardedpoints are above a predetermined prize point value (STEP 1112) orwhether the number of points is greater than the number received byother contestants in the room, division, etc.

[0150] If the contestant is to receive a prize, the server 104 notifiesthe contestant. In one embodiment, different prizes are awarded to thecontestant based on the difference between the contestant's awardedpoints and the predetermined prize point value (STEP 1120). The prizemay be a tangible or intangible good. For example, a prize may include,without limitation, a monetary award, a vehicle. a computer, a vacation,and an interview with a potential employer. In a further embodiment, theprize is awarded to charity on behalf of the winner. The prize may bedelivered to the contestant in a variety of ways, such as mailing theprize to the contestant or having the contestant pick up the prize at apredefined location. In another embodiment, the coding competitionadministrator provides capital for one or more prizes.

[0151] In one exemplary embodiment, the prize distribution for a singleround of competition, is based on the division and the rooms. Thecontestants are seeded into rooms based on their rating. The contestantsin the higher (i.e. lower-numbered) divisions and rooms is as shown inTable II. The prize money for first through third positions of eachdivision and room is as follows: Division 1 Room Contestants 1^(st)2^(nd) 3^(rd) 1 9 635 537 456 2 9 382 323 274 3 9 293 248 211 4 9 249211 179 5 10 249 211 179 6 10 227 192 163 7 10 211 179 152 8 10 196 166141 9 10 183 155 132 10 10 172 145 123 11 10 165 140 119 12 10 158 133113

[0152] Division 2 Room Contestants 1^(st) 2^(nd) 3^(rd) 1 10 91 77 65 210 86 73 62 3 10 81 69 58 4 10 73 62 52 5 10 69 58 49 6 10 66 56 48 7 1063 53 45 8 10 60 51 43 9 11 62 52 45 10 11 59 50 43 11 11 51 43 36

[0153] TABLE II Unrated Room Contestants 1^(st) 2^(nd) 3^(rd) 1 10 100 00 2 11 0 0 0 3 11 0 0 0

[0154] The server 104 may also provide “pick-up” competitions, wherecontestants communicate with the server 104 at any time and eachcontestant pays a competition fee to compete. The competitionadministrator can then distribute a portion of the total amountcollected (e.g., total amount collected less a percentage asadministration fee) to the winner of the pick-up competition.

[0155] In some embodiments, a series of coding competitions are used toidentify premier coders. Contestants participate in rounds, with thehigh scorers for each round participating in the next round. Thisenables initial participation by a large number of contestants, with thegoal of identifying the premier coders in that large number ofcontestants.

[0156] In one exemplary embodiment, a national tournament is held in theUnited States. In this embodiment, early rounds of the competition takeplace over the Internet, with each of the contestants participating fromhome. Regional rounds take place in a central physical location for thatregion, and a national round is hosted in a large vacation-type settingsuch as Foxwoods in Ledyard, Conn.; Las Vegas, Nev.; or Orlando, Fla. Asponsor of the coding competition may receive in-person access tocontestants during the higher rounds, in the form of contactinformation, meals, meetings, and requests/performance of services (e.g.programming, publicity, etc.).

[0157] Referring to FIG. 12, in one embodiment, contestants can progressfrom a first elimination round 1204 through the national championship1220 of a competition series. A predetermined number of contestants areinvited to compete in the first elimination round 1204, based on ratingand/or other factors. In each round, the server 104 divides contestantsinto groups each having a predetermined number of contestants. Eachcontestant in the group competes against the other contestants in thatgroup in a virtual room. In another embodiment, the server 104 populatesa room with a predetermined number of contestants who have similarratings. For example, a first virtual room may have the ten contestantswith the highest ratings, while another virtual room has ten contestantswho have never entered a coding competition before. This sort of roomseeding enables fairer competition between contestants. Further, in oneembodiment prize money is awarded to one or more contestants in eachvirtual room who, for instance, accumulate the highest point value forthat room. In some embodiments, an effort is made to allocatecontestants to virtual rooms by their geographic location, so that theycan participate in a regional competition with others from the sameregion.

[0158] In this embodiment, as the stakes increase (e.g., the amount ofprizes awarded) from the first preliminary round 1208 to a nationalround 1220, so does the level of competition. Each round of thecompetition involves coding, challenge and/or testing. Some of thecontestants in the first preliminary round 1208 are eliminated duringthe first preliminary round 1208, and so fewer number of contestantsparticipate in the second preliminary round 1212. Likewise, still morecontestants are eliminated as the process continues into a regionalround 1216, in which, in one embodiment, the clients 108 and server 104are located at one central physical location for that region. Finally, apredetermined number of contestants having the highest points in theregional round 1216 enter a national round 1220. The national round 1220preferably takes place in one physical location (e.g., a vacation-typesetting, or at a large computer conference). It is intended that thisembodiment is exemplary, and that other embodiments might includedifferent numbers of rounds, contestants, and locations.

[0159] Referring to FIG. 13, an example screen shot of an embodiment ofa virtual “lobby” area 1300 is shown. In one embodiment, the lobby 1300is the main web page displayed on the client 108 during a codingcompetition. The lobby 1300 includes a status panel 1304 which contains,for instance, a contestant's login name, ranking, and status (i.e.,whether the contestant is coding or challenging). The lobby 1300 alsoincludes a main menu 1308 providing functions that can be used at anytime, such as a search function to search for a particular contestant.The lobby 1300 further includes an options area 1310, providing optionsto tailor the display of the lobby 1300. An active contest area 1312 isan area displaying the current active competitions (e.g., a practicecompetition, an actual competition). The lobby 1300 also includes a helparea 1316 for a user to retrieve help on a particular item, and acontest timer display 1320 which displays a relevant time related to thecoding competition. For instance, the contest timer display 1320 candisplay a start timer which counts down to the start of the nextcompetition. A legend key 1322 displays the contestants competing in thenext coding competition. The lobby 1300 also includes a chat area 1324which enables contestants in the virtual room to communicate with eachother. The lobby 1300 also enables a contestant to search for anothercontestant via a search box 1328. Additionally, a contestant informationwindow 1332 enables a contestant to find out certain information aboutanother contestant.

[0160] Referring to FIG. 14, in the embodiment of FIG. 13, a contestantcompeting in the coding competition selects one coding problem fromseveral coding problems that the server 104 transmits to each client108. An exemplary coding problem selection screen 1400 includes alisting 1404 of all virtual rooms that are active. The screen 1400 alsoincludes a tools window 1408. The coding problem selection screen 1400additionally includes a selection window 1412 where a contestant choosesa coding problem based on a displayed point value for that codingproblem. Once a contestant selects a coding problem, the server 104displays a coding window to enable a contestant to create computer codeto solve the coding problem.

[0161] Referring to FIG. 15, an exemplary embodiment of a coding window1500 includes a problem information window 1504 to display informationabout the coding problem, such as the method name, class name, returntype, and argument types. The coding window 1500 also includes a problemstatement window 1508 to display the coding problem. The coding window1500 additionally includes a coding area 1512 in which the contestantwrites computer code attempting to solve the coding problem. Moreover,the coding window 1500 includes an action bar 1516, where the action bar1516 enables the contestant to perform various actions on the computercode, such as testing the computer code, saving the computer code, andsubmitting the computer code to the server 104.

[0162] Referring to FIG. 16, in one exemplary embodiment, aftercompleting the coding stage, the server 104 executes the challenge stageand displays a challenge window 1600. The challenge window 1600 includesa viewing window 1604 that enables a contestant to view othercontestants' computer code. The challenge window 1600 also includes achallenge submission window to enable the submission of test data.

[0163] In one embodiment, the server 104 assigns teams for a teamcompetition. This assignment may occur, for example, before acompetition begins, after a predetermined number of rounds in acompetition, or after one competition is complete and for futurecompetitions. In further embodiments, the server 104 assigns teamsrandomly, based on previous ratings, or based on previous pointsreceived by one or more contestant. In other embodiments, contestantsform teams. In yet another embodiment, the server 104 elects at leasttwo contestants to be team captains and the team captains selectcontestants to be members for their team. Furthermore, the server 104can allow a contestant to be traded from one team to another team.Examples of a trade that one team (e.g., the team captain) can make forone contestant include, but are not limited to, a member of anotherteam, points, and prizes.

[0164] A team competition can involve seasons, where teams compete forthe best record or point total over a predetermined period of time.Moreover, the server 104 may include separate divisions for teams. Inthis embodiment, teams with an average rating above a predetermined teamrating, for instance, are placed into a higher division. The server 104may also provide a “handicap” for a team based on average ratingadvantage. In one embodiment, the server 104 provides this handicap byawarding contestants on the team with handicap points before thecontestant begins or during the contestant, without the contestantanswering a coding problem.

[0165] In some embodiments, the server 104 enables corroboration betweencontestants on the same team, such that each contestant on a team canview the work and/or progress of other team members. In anotherembodiment, the server 104 aggregates the scores of the various teammembers (i.e., disallowing collaboration) to arrive at a team score. Inone embodiment, the server 104 provides a team competition in which someor all team members have to use the computer code created by other teammembers of the team to create the final computer code for submission.

[0166] In another embodiment, the contestants compete with respect tohardware logic design. The server 104 provides a coding problem whichrequests each contestant to design of electronic circuitry and submitthe computer code for the circuit design when complete. The codedcircuit can then be challenged using test data as input into asimulation of the circuit. The execution of the code, then, is executionof a simulation of the circuit specified by the code. The server 104 canthen award points to the contestant based on, for instance, thecircuit's efficiency (e.g. number of components, projectedimplementation cost, maximum input and response speed, etc.),correctness in its response relative to a predetermined solution, andsuccess with respect to the test data. Simple examples of a codingproblem for a circuit would be design of an eight-bit a multiplicationcircuit (i.e., a circuit which multiplies two numbers) using specifiedlogic gates in the digital hardware domain, or the design of a low-passfilter in the analog domain. Likewise, the system and methods of theinvention can be applied to coding in other computer and non-computerdisciplines. If the result can be coded, and then the code executed orsimulated then the principles described above apply. For example, in thefield of civil engineering, with the proper tools, a portion of a bridgedesign (e.g. support truss) can be coded and the coded design can betested with test stresses during simulation. In the field ofaeronautical engineering, an airplane wing elevator design can be codedand the result subjected to various inputs (e.g. shifts in wind, angle,etc.) during simulation. Thus, with the appropriate coding tools,compilers, and execution environment, the method and systems describedcan be extended to coding in various disciplines.

[0167] Variations, modifications, and other implementations of what isdescribed herein will occur to those of ordinary skill in the artwithout departing from the spirit and the scope of the invention asclaimed. Accordingly, the invention is to be defined not by thepreceding illustrative description but instead by the spirit and scopeof the following claims.

What is claimed is:
 1. A method for providing a coding competitioncomprising: (a) transmitting a coding problem to contestants over acommunications network; (b) receiving computer code developed by atleast one of the contestants in response to the transmitted codingproblem over the communications network; (c) determining a response ofthe received computer code to test data submitted by a party other thanthe contestant; (d) evaluating the determined response of the computercode to the test data.
 2. The method of claim 1, wherein step (b)further comprises receiving computer code from a plurality ofcontestants.
 3. The method of claim 1, wherein the evaluating step (d)comprises: determining a reference response of reference code to thetest data; and comparing the determined response of the receivedcomputer code to the test data to the determined reference response ofthe reference code to the test data.
 4. The method of claim 1 whereinthe evaluating step (d) comprises evaluating at least one of thecorrectness of the determined response, the accuracy of the determinedresponse, and the speed of the determined response.
 5. The method ofclaim 1 further comprising the step of awarding points to the at leastone of the contestants based on the evaluation of the determinedresponse of the computer code to the test data.
 6. The method of claim 5further comprising assigning a rating to the at least one of thecontestants based on the points awarded to the contestant.
 7. The methodof claim 1 further comprising, prior to the step of transmitting thecomputer coding problem, requiring contestants to take a qualifyingtest.
 8. The method of claim 7 wherein the step of requiring completionof the qualifying test comprises a determined qualifying test score forthe contestant greater than a predetermined value.
 9. The method ofclaim 8 further comprising, prior to the transmitting step, incrementinga competition counter for each contestant who has obtained thedetermined qualifying test score greater than the predetermined value.10. The method of claim 9 further comprising closing the codingcompetition to other competitors even if the other competitors haveobtained the determined qualifying test score greater than thepredetermined value when the competition counter for the codingcompetition reaches the predetermined value.
 11. The method of claim 1further comprising, prior to the step of transmitting the computercoding problem, the step of assigning the contestant to a division. 12.The method of claim 11 wherein the step of assigning the contestant to adivision further comprises assigning the contestant to one of: (i) afirst division for contestants who have previously attained a ratinggreater than a first predetermined division rating; and (ii) a seconddivision for contestants who have a rating substantially equal to zeroor a rating below a second predetermined division rating.
 13. The methodof claim 12 wherein the first predetermined rating and the secondpredetermined rating are the same rating.
 14. The method of claim 12wherein the method further comprises assigning the contestant to one ofthe first division, the second division, and a third division, the thirddivision for contestants having a rating greater than the secondpredetermined rating and less than the first predetermined rating. 15.The method of claim 12 wherein the step (a) of transmitting a computercoding problem further comprising transmitting less difficult codingproblems to a contestant assigned to the second division relative to acontestant assigned to the first division.
 16. The method of claim 1wherein the step (c) of determining a response of the received computercode to test data submitted by a challenging party other than thecontestant further comprises receiving the test data from one of thecontestants.
 17. The method of claim 1 wherein the step (c) ofdetermining a response of the received computer code to test datasubmitted by a challenging party other than the contestant furthercomprises receiving the test data from a spectator.
 18. The method ofclaim 1 wherein the step (c) of determining a response of the receivedcomputer code to test data submitted by a challenging party other thanthe contestant further comprises generating the test data.
 19. Themethod of claim 1 wherein the step (c) of determining a response of thereceived computer code to test data submitted by a challenging partyother than the contestant further comprises determining a response ofthe computer code to a predetermined test data.
 20. The method of claim1 wherein the communications network is the Internet.
 21. The method ofclaim 1 wherein the communications network is an intranet.
 22. Themethod of claim 1 wherein the communications network is a local areanetwork.
 23. The method of claim 1 wherein the communications network isa wide area network.
 24. The method of claim 1 wherein computer code issource code, and wherein the method further comprises, prior to step(c), the step of compiling the source code thereby producing executablecode; and wherein step (d) comprises determining the response ofcompiled executable code to the test data.
 25. The method of claim 1wherein step (b) further comprises receiving computer code written inthe C++ programming language.
 26. The method of claim 1 wherein step (b)further comprises receiving computer code written in the C# programminglanguage.
 27. The method of claim 1 wherein step (b) comprises receivingcomputer code that is a model that can be forward engineered into amachine-readable format for testing purposes, and wherein step (c)comprises testing the machine-readable format of the model.
 28. Themethod of claim 1 wherein step (b) further comprises receiving computercode that is interpreted code.
 29. The method of claim 28 wherein step(b) further comprises receiving computer code written in the Javaprogramming language.
 30. The method of claim 1 wherein step (b) furthercomprises receiving computer code comprising an engineering design, andwherein step (c) comprises simulating the engineering design.
 31. Themethod of claim 30 wherein the engineering design comprises anelectronic circuit design.
 32. The method of claim 1 wherein the methodsteps (a)-(d) are repeated first in a practice session and second in acompetition.
 33. The method of claim 1 wherein: step (a) comprisestransmitting a coding problem to at least two teams of contestants; step(b) comprises receiving computer code developed by at least one of theteams of contestants; and step (c) comprises determining a response ofthe received computer code to test data received from a party not partof the team that developed the computer code.
 34. The method of claim 1wherein the transmitting step (a) takes place upon request by two ormore available contestants.
 35. The method of claim 1 wherein thetransmitting step (a) takes place at a predetermined time.
 36. A methodfor evaluating a skill level of a contestant comprising: (a)electronically communicating a coding problem to contestants during acompetition; (b) electronically receiving computer code in response tothe coding problem from one of the contestants during the competition;(c) automatically evaluating the received computer code; (d) awardingpoints to the contestant based on the received computer code; and (e)determining a rating for the contestant based on the number of pointsawarded to the contestant.
 37. The method of claim 36 wherein step (e)further comprises determining an average rating for the competitionbased on previous ratings of the contestants and a total number ofcontestants receiving the coding problem.
 38. The method of claim 37wherein step (e) further comprises determining a competition factor forthe competition based on at least one of previous ratings ofcontestants, a total number of contestants receiving the coding problem,and volatility of the ratings of the contestants.
 39. The method ofclaim 38 wherein step (e) further comprises determining a skill for acontestant based on a previous rating of the contestant.
 40. The methodof claim 39 wherein step (e) further comprises determining a deviationvalue for a contestant relative to other contestants based on thevolatility of the ratings of the contestants.
 41. The method of claim 40wherein step (e) further comprises estimating a probability of awardinga contestant more points than other contestants in the competition. 42.The method of claim 41 wherein step (e) further comprises calculating atleast one of an expected performance, an expected rank, and an actualperformance of a contestant.
 43. The method of claim 42 wherein step (e)further comprises calculating at least one of a new volatility and a newrating of a contestant.
 44. The method of claim 43 further comprisingdetermining a maximum rating increase for the contestant.
 45. The methodof claim 44 further comprising determining a weighting effect of thecompetition for the contestant.
 46. The method of claim 36 wherein thestep of determining of a rating for the contestant further comprisesadjusting the rating in smaller increments as the frequency that thecontestant has been rated increases.
 47. A method for providingcontestant information to a potential employer, comprising: receivingcontestant information from contestants for entry into on-line codingcompetitions; (a) providing an on-line coding competition betweencontestants; (b) assigning a rating to each contestant based onperformance in the on-line coding competition; and (c) providing atleast some contestant information and the assigned ratings of at leastone contestant to a potential employer of the contestant.
 48. The methodof claim 47 wherein step (c) further comprises awarding points to thecontestant based on performance in the on-line competition.
 49. Themethod of claim 47 wherein the contestant information further comprisesat least one of home address, phone number, work address, e-mailaddress, professional experience, work history, and certain demographicinformation of the contestant.
 50. A method for awarding a prize to acontestant comprising: (a) providing an on-line coding competition amongcontestants; (b) awarding a point value to each contestant based onperformance in the on-line coding competition; and (c) awarding at leastone prize to at least one contestant in response to the awarded pointvalue.
 51. The method of claim 50 wherein the method comprises awardingprizes to contestants who have been awarded a point value above apredetermined prize point value, wherein the prize awarded depends onthe points awarded with respect to the predetermined prize point value.52. The method of claim 50 wherein the prize awarded is at least one ofa monetary award, a tangible good, an interview with a prospectiveemployer, and an employment offer.
 53. A method for advertising in acoding competition comprising: (a) receiving contestant information fromcontestants for entry into on-line coding competitions; (b) providing atleast one on-line coding competition between contestants; (c) providingeach contestant with at least one advertisement for viewing during theat least one competition; (d) tracking the viewing time of the at leastone advertisement by each contestant; and (e) providing at least somecontestant information and the viewing time of the at least oneadvertisement to an advertiser.
 54. The method of claim 53 furthercomprising the step of assigning a rating to each contestant based onperformance in the on-line coding competition.
 55. The method of claim54 wherein the contestant information providing step (e) furthercomprises providing the contestant rating.
 56. The method of claim 53wherein the contestant information is provided in the aggregate in thecontestant information providing step (e).
 57. The method of claim 53wherein the step of providing each contestant with at least oneadvertisement for viewing during the at least one competition comprisesselecting an advertisement for viewing by at least one contestant inresponse to contestant information.
 58. An apparatus for providing acoding competition, the apparatus comprising: (a) a web server forcommunicating with a web browser used by a contestant to provideinformation about a coding competition; (b) a client interface servercommunicating with the client software used by the contestant, theclient interface server enabling transmission of a coding problem to theclient software and reception of at least one of a client request andcomputer code in response to the coding problem; (c) an applicationserver in communication with the web server and the client interfaceserver to interpret at least one of the client request, the competitionrequest, and the computer code; and (d) a contest server incommunication with the application server to at least one of compile thecomputer code, execute the computer code, test the computer code, andcompute a rating for the contestant.
 59. The apparatus of claim 58further comprising a database server in communication with theapplication server to retrieve data in response to at least one of theclient request. the competition request, and the computer code.
 60. Anapparatus for determining a coding skill level of a coding contestant,the apparatus comprising: (a) a client-server interface for providing acompetition communication to a contestant for entry into a codingcompetition with a plurality of contestants; (b) a receiver forreceiving a request from the contestant; and (c) a contest server layerin communication with the receiver to determine a coding skill level ofthe contestant by at least one of validating the request and determininga rating of the contestant with respect to the plurality of contestants.61. The apparatus of claim 60 further comprising an application serverlayer in communication with the receiver to interpret the response. 62.The apparatus of claim 61 wherein the application server layer furthercomprises a message queue processor in communication with the receiverto queue the request for operation by the contest server layer.
 63. Theapparatus of claim 62 wherein the message queue processor is incommunication with the contest server layer to transmit the request tothe contest server layer.
 64. The apparatus of claim 60 wherein thereceiver further comprises a web server layer.
 65. The apparatus ofclaim 60 wherein the receiver further comprises a client interfaceserver layer.
 66. The apparatus of claim 65 wherein the client interfaceserver layer further comprises an interface connection providing aninterface to the client-server interface.
 67. The apparatus of claim 60wherein the client-server interface further comprises at least one of aweb browser and client software.
 68. The apparatus of claim 60 whereinthe client interface server layer further comprises an interface servermanager in communication with the interface connection to manage atleast one of the allocation and operation of communications with theinterface connection.
 69. The apparatus of claim 60 wherein the clientinterface server layer further comprises an interface server process incommunication with the interface connection to provide communicationswith the application server layer.
 70. The apparatus of claim 60 whereinthe competition communication further comprises a coding problem. 71.The apparatus of claim 60 wherein the request further comprises aresponse to the coding problem.
 72. The apparatus of claim 60 whereinthe application server layer further comprises a servlet engine incommunication with the web server layer.
 73. The apparatus of claim 72further comprising a database server layer in communication with theapplication server layer.
 74. The apparatus of claim 72 wherein theapplication server layer further comprises a data access layer to accessdata in response to at least one of the request and the interpretationof the request.
 75. The apparatus of claim 74 wherein the databaseserver layer is in communication with the data access layer.
 76. Theapparatus of claim 60 wherein the contest server layer further comprisesat least one of a rating engine, a code compilation engine, and a codetesting engine.
 77. The apparatus of claim 76 wherein the contest serverlayer is in communication with the message queue processor.